How to Group Crashes Effectively: Comparing Manually and Automatically Grouped Crash Dumps

Crash dumps have become an important source for software developers to learn quality issues in released software. Since a same bug can be repeatedly triggered by different users, an overwhelming number of crash dumps are returned daily. Techniques for automatically grouping crash dumps are mostly based on call stacks captured at crash sites; although fast, they can incorrectly group irrelevant crash dumps or miss related crash dumps. The goal of this paper is to compare manually and automatically grouped crash dumps and discover grouping criteria that are effectively used in manual diagnosis but lacking in automatic tools, from which we potentially enable more precise and capable crash diagnostic tools. In our study, we compared a total of 1,550 groups and 30,431 crash dumps from 5 Mozilla applications. We find that 1) call stacks are more dissimilar from each other in manual groups, as besides matching call stacks, developers frequently link multiple sources, such as reproduce steps and revision histories, for grouping crashes; 2) while automatic tools focus on grouping crashes based on the same root cause, developers also correlate crash dumps across different versions of programs or even different applications for related root causes; 3) both automatic and manual approaches are imprecise, but the two make different types of mistakes; and 4) for small applications, developers’ domain knowledge on code works more effectively than the automatic approach for correlating crashes. From the study, we learn that to more effectively group crashes, our future tools should 1) enable multiple criteria and explore diverse uses of crash dump groups for prioritizing and fixing bugs, 2) correlate multiple sources of information or even multiple applications, and 3) uncover inherent relations between symptoms and source code. Keywords-Crash Dumps, Call Stacks, Grouping, Similarity